REMARKS 

The present Amendment C is being filed concurrently with an RCE. 

Upon entry of the present Amendment C, the claims in the application are claims 1-15, of 
which claims 1, 3, 6, 7, 9 and 10 are independent. Claims 1-3 and 6-12 are amended herein, and 
new claims 13-15 are added. 

Initially, applicant would like to thank Examiner Repko for the helpful and courteous 
telephonic interview he conducted with the applicant's representatives August 9, 2006. An 
argument was presented that the camera disclosed in the Kolb reference does not correspond to the 
camera recited by the applicant in claim 1, but instead refers to a physical camera upon which the 
disclosed camera model is based. The camera model is used to form an accurate 2-D image of a 3-D 
model. It is the applicant's position that, although the system of Kolb models a physical camera, the 
physical camera is not the same camera that is used to acquire the picture used in a composite image. 
That is, the image of the 3-D model obtained (synthesized) with the model of Kolb does not 
consider defining lines of sight so that the lines of sight conform to rays of light corresponding to 
those of the picture with which the rendered image is to be combined. The Examiner disagreed with 
our arguments, citing Kolb at col. 2, lines 39-40 and at col. 5, lines 37-40 to substantiate his position 
that Kolb expressly recites that the location of the model camera relative to the radiant scene is 
chosen to mimic the physical camera's position in the acquired image. No agreement was reached. 

The above-identified Office Action has been reviewed, the applied references carefully 
considered, and the Examiner's comments carefully weighed. In view thereof, the present 
Amendment C is submitted. It is contended that by the present amendment, all bases of rejection 
set forth in the Office Action have been traversed and overcome. Accordingly, reconsideration 
and withdrawal of the rejection is respectfully requested. 
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Claim Rejections - 35 USC $103 

At item 3 of the Office Action (page 2), the Examiner rejected claims 1, 6, 7 and 10 
under 35 USC §103 (a) as being unpatentable over Noyama et al. (US 5,594,850) in view of 
Kolb et al. (US 6,028,606). In his rejection, the Examiner states that Noyama discloses a 
method for compositing a computer graphics (CG) image and a picture taken by a camera (col. 4, 
lines 41-44), comprising defining a 3-D model, a viewpoint (col. 6, lines 26-29) and a plane of 
projection, in a space established on a computer (col. 6, lines 29-34); tracing lines of sight 
extending from the viewpoint through the plane of projection and the three dimensional model to 
obtain attributes of portions of the three dimensional model corresponding to the projection 
pixels (col. 7, line 66-col. 8, line 3), thereby forming a 2-D image of the 3-D model on the plane 
of projection (col. 8, lines 39-41); and superposing the 2-D image 20 on the picture 30 to 
generate a composite image 50 (col. 10, lines 22-29, Fig. 8). 

Further, the Examiner states that Noyama et al. do not expressly disclose defining lines of 
sight extending from the viewpoint to projection pixels on the plane of projection so that each of 
the lines of sight conform with a ray of light incident on a pixel corresponding thereto of the 
picture taken by the camera. However, the Examiner states that Kolb discloses a function to 
define lines of sight extending from the viewpoint to projection pixels on the plane of projection 
so that each of the lines of sight conforms with a ray of light incident on a pixel corresponding 
thereto of the picture taken by the camera (col. 12, lines 57-59). The Examiner considers it 
obvious to combine Noyama with Kolb to obtain the invention as specified in claim 1 since Kolb 
teaches that it is desirable to seamlessly merge acquired imagery with synthetic imagery, and 
important to compute the synthetic imagery using a camera model that closely approximates the 
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real camera and lens system. 
Applicant's Response 

Upon review of Noyama, the applicant finds that Noyama discloses a method of image 
simulation for creating a composite still or single image by combining a CG image 20, formed 
from a 3-D CG model 10 consisting of an object shape model 12, with a natural image 30. The 
method of Noyama generally includes segmenting the CG image 20 in order to create a mask 
image 22; defining a mask attribute table 33 for the mask image by data transformation; creating 
an intrinsic image 28 (consisting only of the pixel values needed for the region to be color 
transformed) by data transformation; cutting a portion from the CG image using the image mask; 
and combining the cut portion of the CG image with the natural image in order to form the 
composite image (col. 4, line 31-col. 5, line 7, Fig. 1, and col. 7, lines 1-2). 

Noyama describes obtaining the 2-D CG image 20 by projecting the 3-D model 12 onto a 
projection plane using a parallel projection method such that the 2-D image is created 
corresponding to a shadow of the shape model 12 on the projection plane 240 (Fig. 4). The 
projection (ie, shadow) is used to form the mask image 22. In particular, Noyama discloses that 
in conventional computer graphics, images are generally created by tracing rays from the light 
source 14 to determine the colors on the screen, but in the invention of Noyama, the tracing 
direction is reversed such that the rays are traced starting from the eye (viewpoint) until the color 
of the pixel is determined. That is, the ray tracing starts from the eye 16 and continues along a 
straight line connecting the eye 16 and the pixel being processed on the projection plane until 
striking an object 12. Attributes of 3-D object at this intersection are stored in the mask and 
intrinsic image attribute tables (col. 7, line 45- col. 8, line 12). 

Thus the mask image 22 does not have the details of the CG image 20 related to color 
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and pixel intensity, but merely comprises divided or segmented regions, the regions being masks 
respectively identified with mask ID numbers for later use in producing the composite image. 

A natural image 30, obtained from a camera, scanner, or the like, consists solely of color 
information for each pixel and a human operator carries out pre-arranged processing for cutting 
out the object (ie, the tree 66 in Fig. 6) from the natural image (tree plus background), creating a 
mask image 32 (tree only) from the natural image 30. This information is used to manually fill a 
mask attribute table for the cutout object. In the disclosure of Noyama, the CG image 20 is 
created from the 3-D image along with simultaneous automatic generation of transformation data 
for later use in image simulation, and the transformation data is compatible with the pixel color 
data in the natural image 30, whereby it is possible to create a composite image 50 by combining 
the CG image 20 (that is, the transformation data) and the natural image 30 (that is, the mask 
image 32 of the natural image) in the simulation section 300. 

As regards the rejection of claims 1, 6-7 and 10, the applicant respectfully disagrees that 
Noyama discloses defining lines of sight so that each of the lines of sight conforms with a ray of 
light incident on a pixel corresponding thereto of the picture taken by the camera, as claimed. 
As discussed above, in the method of Noyama, the rays are traced starting from the eye 
(viewpoint) until the color of the pixel is determined. That is, the ray tracing starts from the eye 
16 and continues along a straight line connecting the eye 16 and the pixel being processed on the 
projection plane until striking an object 12. Attribute data of the location of intersection of the 
ray with the object is obtained and stored. This procedure is performed on the 3-D model to 
generate the 2-D CG image, and thus is performed completely independently of, and without 
reference to, the natural image 30, and without consideration of how the natural image is formed 
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(ie, camera angle relative to the ground, lighting, etc). Thus, Noyama does not define lines of 
sight in consideration of the picture taken by the camera, or natural image. In the disclosure of 
Noyama, neither the natural image 30 or the data associated with the natural image are used in 
generation of the 2-D CG image, but instead, after the 2D image is formed, information from the 
2D image is combined with the natural image to form the composite image 50. In contrast, the 
applicant discloses pre-determining the directions of rays of light incident on pixels of an image 
plane of the camera and correlating them to pixel positions in a frame of the picture taken by the 
camera. The lines of sight for tracing for use in obtaining attributes (ie, color) of the 3-D model 
on the projection pixels are made to conform to a ray of light incident on the pixel of the image 
plane of the camera corresponding to the projection pixel. 

As further regards the rejection of claims 1, 6-7 and 10, the applicant respectfully 
disagrees that Noyama discloses tracing the lines of sight extending from the viewpoint through 
the plane of projection and [to] the three-dimensional model to obtain attributes of portions of 
the three-dimensional model corresponding to the projection pixels. Rather, Noyama teaches use 
of a parallel projection method in which the 2-D image is created corresponding to a shadow of 
the shape model 12 on the projection plane 240 (Fig. 4, col. 7, lines 14-33). In Fig. 4, the sight 
lines from the viewpoint, or eye 16 (not shown), are shown as dotted lines, and the location of 
the eye 16 with respect to the model 12 is not expressly identified. Since the sight lines of Fig. 4 
extend from the object 12 to the plane 240, it is presumed that the eye 16 is positioned to the left 
of the object. Thus, Noyama does not disclose tracing the lines of sight extending from the 
viewpoint through the plane of projection, as claimed. 

Modification of Noyama by the disclosure of Kolb fails to cure the deficiencies of 
Noyama as described above. Upon review of Kolb, the applicant finds that Kolb discloses a 



15 



camera simulation system (ie, camera-based models) to synthesize a 2-D image from data 
representing a 3-D scene for the purpose of rendering images which are more realistic and 
closely resemble images created with a camera system. Thus Kolb provides a method of 
modeling a physical camera, including specified lens, aperture configuration, shutter setting, and 
film surface. 

In the rejection, the Examiner states that Kolb discloses a function to define lines of sight 
based on the "calibration data" for a camera system, and "defines lines of sight extending from 
the viewpoint to projection pixels on the plane of projection so that each of the lines of sight 
conforms with a ray of light incident on a pixel corresponding thereto of the picture taken by the 
camera" (col. 12, lines 57-59). 

The applicant respectfully disagrees that Kolb discloses a function to define lines of sight 
based on "calibration data" for a camera system, as put forth by the Examiner in the rejection. 
Kolb does not disclose use of any type of calibration data when defining lines of sight, but 
instead discloses applying a ray tracing algorithm (col. 6, lines 6-47) in order to construct rays 
from a point x' in the pixel area to a point x" in the exit pupil and then from x" to the scene 
data. The applicant agrees that Kolb discloses using camera specifications such as lens 
dimensions, indices of refractions, and relative orientations in order to implement ray tracing 
techniques, but the applicant asserts that such specifications do not constitute calibration data. 

In addition, the applicant disagrees that the camera disclosed by Kolb corresponds to the 
camera recited by the applicant in claim 1 . The camera disclosed by Kolb refers to the physical 
model of the camera which is acquiring the 2-D image of the 3-D model, whereas the in the 
applicant's claimed invention, the picture taken by the camera refers to the non-CG image, that 
is, the image to be combined with the CG image. 
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In the interview of August 6, 2006, the Examiner stated that the Kolb reference, at col. 2, 
lines 39-40 and at col. 5, lines 37-40 to substantiate his position that Kolb expressly recites that the 
location of the of model camera relative to the radiant scene is chosen to mimic the physical 
camera's position in the acquired image. 

At col. 2, lines 39-40, Kolb describes the desirability of seamlessly merging acquired 
imagery with synthetic imagery, and the importance of using a computing of the synthetic imagery 
using a camera model that closely approximates the real camera and lens system. Although the 
background section (col. 2, lines 39-40) of Kolb clearly discusses the general intent to combine the 
2-D image generated using the inventive camera model (synthetic imagery) with images acquired 
from a camera (acquired imagery) to form a composite image, Kolb is silent as to the acquired image 
and to any influence or consideration of the acquired image when forming the synthesized image . 
That is, there is no suggestion that Kolb considers defining lines of sight in the synthetic image so 
that the lines of sight conform to rays of light corresponding to those of the acquired image, as 
specifically claimed. 

At col. 5, lines 37-40, Kolb states the location of the model camera relative to the radiant 
scene is chosen to mimic the physical camera's position in the acquired image. The Examiner 
considers "the physical camera" discussed herein to refer to a physical camera which generates an 
acquired image for use in merging with a synthetic image generated by the camera model. However, 
upon careful review of the disclosure of Kolb, the applicant disagrees. In every instance in which 
Kolb uses the term "physical camera", this term is used to represent the idealized structure upon 
which the camera model is based and the functions of which the camera model is attempting to 
mimic (for example, col. 3, lines 27-39, and at col. 14, lines 36-39). Thus Kolb's physical camera is 
a different camera from the claimed camera which is used to obtain a picture with which the 
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synthesized 2-D image is to be combined. 

Although the applicant disagrees that Noyama as modified by Kolb make obvious the 
applicant's invention for the reasons discussed above, in order to promote the prosecution of the 
application, the applicant has amended the independent claims herein to more clearly distinguish 
the applicant's invention from that of the cited prior art. 

In particular claims 1, 6, 7 and 10 are amended herein to more clearly recite that the 
model is defined in accordance with the conditions in which the picture is taken. This subject 
matter is disclosed in the specification at page 17, lines 1-11, and thus no new matter is added. 

This subject matter is not suggested by Noyama or Kolb, alone or in combination. 
Noyama docs not disclose the particulars of how the model is defined, and merely discloses at 
col 6, lines 25-29, that a CG model 10 consists of an object shape model 12 made up of object 
shape and surface attributes. Similarly, at column 4, lines 52-53, Kolb merely discloses storing 
3-D radiant scene data 20 in a computer memory using standard techniques. 

The present invention is derived from the precise distance measurement technique using 
a stereo camera. The applicant believes that persons skilled in the art would appreciate that the 
Kolb premise of "when given the manufacturer's specifications of the physical camera's lenses, 
including the dimensions and indices of refraction of its lenses, stops, and shutter characteristics, 
the location of the film surface relative to the lens system, and the orientation of the camera 
within the scene" (col. 3, lines 28-36, cited by the Examiner in item 36 of Response to 
Arguments section of the Office Action) is very difficult to establish under the present 
circumstances because of the unwillingness of the manufacturers to disclose the specifications, 
the differences among individual cameras and lenses which cannot meet the required precisions, 
and so forth. Accordingly, in order to produce a 2-D image from a 3-D model so that the 2-D 



18 



image is accurately and seamlessly composited with a picture taken by a camera, the conditions 
in which a picture to be combined with a CG image is taken by a camera is taken into 
consideration according to the present invention. 

At item 13 of the Office Action, the Examiner has rejected claims 2-4, 8-9 and 11-12 
as unpatentable over Noyama et al in view of Kolb et al, and in further view of Benjamin 
Mora et al. "A New Object-Order Ray-Casting Algorithm," October 27, 2002, Proceedings 
of the Conference on Visualization '02 ("Mora"). In the rejection of claims 2, 8, 1 1 and 12, 

the Examiner states that Noyama discloses the limitations of claim 1, but does not disclose a 
calibration table correlating pixel positions and directions of rays of light. Kolb discloses a 
function to define lines of sight based upon the "calibration data" for a camera system (col. 12, 
lines 57-59), but does not disclose using a table to store and access the rays computed using the 
lens simulation function. The Examiner states that Mora stores pre-computed ray data 
corresponding to pixel positions, and discloses a table having first data and second data 
correlated with each other, the first data concerning positions of pixels, the second concerning 
ray data. The Examiner considers it obvious to modify Noyama/Kolb to store the pixel to ray 
correspondences, as well as directions and positions of rays in a table as taught by Mora since 
this permits avoidance of redundant computation during rendering, and pre-computation and 
storage of results is well-known in the art. 
Applicant's Response 

The applicant respectfully disagrees with the rejections of claims 2, 8, 1 1 and 12 for the 
reasons stated above with respect to claims 1, 6, 7 and 10 discussed above, from which these 
claims respectively depend. The applicant disagrees with the Examiner's assertion that Kolb 
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discloses, at col. 12, lines 57-59, a function to define lines of sight based upon "calibration data". 
Kolb does not disclose use of any type of calibration data when defining lines of sight, but 
instead discloses applying a ray tracing algorithm (col. 6, lines 6-47) in order to construct rays 
from a point x' in the pixel area to a point x" in the exit pupil and then from x" to the scene 
data. The applicant agrees that Kolb discloses using camera specifications such as lens 
dimensions, indices of refractions, and relative orientations in order to implement ray tracing 
techniques, but the applicant asserts that such specifications do not constitute calibration data. 

In addition, the applicant disagrees that the disclosure of Mora provides a teaching which 
cures the deficiencies of Noyama/Kolb. 

Mora discloses an algorithm which permits fast and accurate rendering of 3-D space 
using a technique referred to as object-order ray-casting. In the disclosed technique, Mora 
provides orthogonal projection of a hexagon on the image plane, a square made of four 
neighboring pixels is subdivided, and a list of relative coordinates corresponding to the 
projection of the cell is associated with each subdivision to provide the pixel index shown in 
Figure 2a. Mora also discloses using a set of pre-processed rays to find out a ray entry point into 
the cell, and a ray number pointing to the best representative pre-processed ray is assigned to 
every pixel of the projection list to provide the ray index in Figure 2a. 

The applicant disagrees that the pixel index disclosed by Mora corresponds to data 
concerning positions of pixels of the picture taken by the camera, as claimed, since the disclosed 
pixels are related to the 2-D image generated from the 3-D model relative to a viewpoint. In the 
disclosure of Mora, the viewpoint is arbitrary and unrelated to the picture, that is, unrelated to a 
separate image to which the generated 2-D is to be combined. Similarly, Mora discloses use of a 
ray index, corresponding to a ray number pointing to the best representative pre-processed ray. 



20 



The ray index is also unrelated to directions and rays of light incident on pixels of the picture (a 
separate image to which the generated 2-D image is to be combined). In contrast, the applicant 
discloses pre-determining the directions of rays of light incident on pixels of an image plane of 
the camera and correlating them to pixel positions in a frame of the picture taken by the camera. 
The lines of sight for tracing for use in obtaining attributes (ie, color) of the 3-D model on the 
projection pixels are made to conform to a ray of light incident on the pixel of the image plane of 
the camera corresponding to the projection pixel. 

Although the applicant disagrees that Noyama as modified by Kolb and Mora make 
obvious the applicant's invention for the reasons discussed above, in order to promote the 
prosecution of the application, the applicant has amended claims 2, 8, 1 1 and 12 herein to more 
clearly distinguish the applicant's invention from that of the cited prior art. 

In particular claims 2, 8, 1 1 and 12 are amended herein to more clearly recite that the 
second data comprises correction values which correct lines of sight to correspond to directions 
and positions of rays of light incident on the pixels of the picture. This subject matter is 
disclosed at page 13, lines 20-23 of the specification, and thus no new matter is added. None of 
the cited references suggests or discloses performing a correction of the lines of sight. Kolb, who 
discloses a simulated camera (or camera model) that mimics a physical camera upon which the 
model parameters are based, obtains images which avoid the problems associated with images 
obtained using techniques based on a pin-hole camera mode. Therefore, Kolb does not suggest or 
disclose correcting obtained lines of sight. However, the applicant's invention applies the pin hole 
camera model when acquiring a 2-D image from a 3-D model, and then performs a correction to 
permit the acquired 2-D image to appear the same as one obtained with a lensed camera. 
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As regards claims 3 and 9, the Examiner states that Noyama discloses compositing a 
computer graphics image by obtaining lines of sight (col. 9, lines 18-21), generating a 2-D image 
on the plane of projection from the 3-D model (col .8, lines 39-41), and superposing the 2-D 
image on the picture to generate a composite image, but does not disclose basing the line of sight 
calculation on the acquired imagery and corresponding camera. The Examiner further states that 
Kolb teaches this feature, as discussed above for claim 1 . The Examiner notes that Noyama does 
not disclose a calibration storage unit or looking up directions and positions, but that these 
limitations are similar in scope to the limitations found in claim 2, and that the precomputed 
table disclosed by Mora provides a teaching for these limitations, as discussed in the rejection of 
claim 2. In addition, the Examiner states that Noyama docs not expressly disclose an apparatus, 
or units within an apparatus, as claimed, but that implementation of the disclosed method in 
hardware would have been obvious to one of ordinary skill in the art. 
Applicant's Response 

The applicant respectfully disagrees with the rejections of claims 3 and 9 for the reasons 
stated above with respect to the deficiencies of Noyama, Kolb, and Mora. 

Although the applicant disagrees that Noyama as modified by Kolb and Mora make 
obvious the applicant's invention for the reasons discussed above, in order to promote the 
prosecution of the application, the applicant has amended claims 3 and 9 herein to more clearly 
distinguish the applicant's invention from that of the cited prior art. 

In particular claims 3 and 9 are amended herein to more clearly recite that the first data 
comprises positions of pixels of the picture taken by the camera, and the second data comprises 
directions and positions of rays of light incident on the pixels of the picture corresponding to 
optical properties of the camera. This subject matter is disclosed at page 13, lines 20-21 of the 
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specification, and thus no new matter is added. This feature is not disclosed by Mora, who 
discloses using a set of pre-processed rays to find out a ray entry point into the cell, and stores as 
second data a ray number pointing to the best representative pre-processed ray. Clearly, a ray 
number corresponding to the best representative pre-processed ray does not suggest or disclose 
directions and positions of rays of light incident on the pixels of the picture corresponding to 
optical properties of the camera, as now claimed. 

As regards claim 4, the Examiner states that Kolb further discloses the process of tracing 
a ray of light which strikes an image point x' and a displacement from a base point (col. 10, lines 
28-3 1), and that it would have been obvious to store the direction in which a ray strikes on a 
pixel and displacement from a base point as disclosed by Kolb in a table as taught by Mora in 
order to avoid redundant computation. 
Applicant's Response 

The applicant respectfully disagrees with this rejection. The portion of the Kolb 
disclosure cited by the Examiner as corresponding to the claimed limitation is directed to tracing 
a ray through a thick lens, and does not disclose or suggest a direction in which a ray of light 
strikes on a pixel of the picture, and/or a displacement from a base point to an incident light, as 
claimed. 

At item 25 of the Office Action, the Examiner has rejected claim 5 under 35 USC 
103(a) as being unpatentable over Noyama in view of Kolb and Mora, and further in view 
of F.S. Hill, Jr. "Computer Graphics Using OpenGL," May 15, 2000, 2 nd Ed. Prentice Hall 
("Hill"). In the rejection, the Examiner states that the combination of Noyama, Kolb, and Mora 
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does not show storing as "two points in the incident light", but that Hill teaches thinking of a 
vector geometrically as a displacement from one point to another. The Examiner considers it 
obvious to modify Noyama/Ko lb/Mora by the teaching of Hill in order to simplify logic and 
storage by representing the vector with an analogous entity. 
Applicant's Response 

The applicant respectfully disagrees with this rejection since it would not be obvious to 
modify the second data to be represented by two points. The disclosure of Hill is a review of 
vectors and vector analysis, and the cited text indicates that vectors are objects having length and 
direction, and that it is valuable to think of a vector geometrically as a displacement from one 
point to another. Wc disagree that the second data, comprising direction and position of rays of 
light, corresponds to a vector, which represents direction and magnitude of an object. 

Other Matters 

The applicant has added new dependent claims 13-15 herein. Claims 13-15 are directed 
to a recitation which lines of sight are defined using displacement vectors and direction vectors. 
This feature is disclosed in the specification on page 18-lines 17 - page 18, line 8, and thus no 
new matter is added. In addition, this feature is not suggested or disclosed in the cited prior art 
references. 

Conclusion 

In conclusion, applicant has overcome the Examiner's objection and rejections as 
presented in the Office Action; and moreover, applicant has considered all of the references of 
record, and it is respectfully submitted that the invention as defined by each of present claims is 
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patentably distinct thereover. 

Applicant respectfully submits that all of the above amendments are fully supported by 
the original application. Applicant also respectfully submits that the above amendments do not 
introduce any new matter into the application. 

The application is now believed to be in condition for allowance, and a notice to this 
effect is earnestly solicited. 

If the Examiner is not fully convinced of all of the claims now in the application, 
applicant respectfully requests that he telephonically contact applicant's undersigned 
representative to expeditiously resolve prosecution of the application. 

Favorable reconsideration is respectfully requested. 
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